posted 01-09-2008 05:23 PM            

Here is a test for discussion.

I'm curious about the range of numbers we all get when looking at something like this.

Chart 1 page 1

Chart 1 page 2

Chart 1 page 3

Chart 1 page 4

Chart 2 page 1

Chart 2 page 2

Chart 2 page 3

Chart 2 page 4

Chart 3 page 1

Chart 3 page 2

Chart 3 page 3

Chart 3 page 4

r

------------------

"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-09-2008 05:37 PM            

Thanks Ray!

I have a house full of people now (1 yr old's B-Day) but I will score those when I have an idle sec.

Nice!!!

posted 01-09-2008 05:47 PM            

I've been leaving the raw EDA on, but the amplitude on the tracing is turned all the way down. Its darker green and at the top - ignore it. The system will not presently collect raw EDA, unless the channel is "on". Like Lafayette's storage of ascii data, I believe these things should be set to store/on by defaulat - we'll have to keep heckling the equipment people, but they'll get it.

The orange tracing is another movement channel, but its turned down at present. The data are stored normal, only the display is silenced.

r

------------------

"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-09-2008 08:20 PM            

chart 1

P1 -1 0 -1
P2 +1(0) +1(+1) -1(-1)

G +1 +1 +1

C +1 -1 -1
__________________________________
+2 +1 -1

chart 2

8 4 6
P1 1 0 1
P2 1 0 1

G 1 1 0

C 0 1 0
____________________________________
2 2 1

Chart 3
6 8 4
P1 1 1 0
P2 1 1 0

G 1 1 0

C 1 1 O
--------------------------------------------
3 3 0

Spot Totals 5 5 2

Grand total +12

No Opinion (if multiple issue test)

I scored this test with alot less focus than a field test. My 6 year old is crying because his sheets aren't the "cool ones" while I score. I don't take my 6 yr olf into the lab with me typically (lol)

[This message has been edited by stat (edited 01-09-2008).]

posted 01-09-2008 09:19 PM            

R4- +4
R6- +3
R8- +2

total- +9 NDI

However, I did score the charts quickly and not that thoroughly.

------------------
"Truth Seekers"

posted 01-09-2008 10:08 PM            

I1,SR2,C3,R4,C5,R6,C7,R8,I9

SR2,C5,R8,C7,R4,C3,R6,I9

SR2,C7,R6,C3,R8,C5,R4,I9

posted 01-09-2008 10:13 PM            

Its actually a mixed-issues PCSOT maintenance test.

Its AFMGQT with 3 RQs and 3 CQs, and I generally add another N question to the end. I can use it during the test if needed, or simply leave it at the end. My own limited bit of andecdotal wisdom convinces me its helpful to have more than one NQ in a test. I cropped the client info from the beginning of the test - oh, and I can't count, so the questions are in some random order.

I usually score the first 2 spots to the stronger comparison by component.

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-09-2008 10:25 PM         

my CONSERVATIVE scores are as follows:

Chart 1

4 6 8

0 0 0
+1 +1 +1
+1 -1 -1
______________________________________
+2 0 0

0 0 0
+1 +1 +1
+1 +1 +1
________________________________________
+2 +2 +2

0 0 0
0 +1 +1
+1 +1 +1
_______________________________________
+1 +2 +2

TOTAL Spot Scores (again very conservative)

4 6 8

+5 +4 +4

The Pnumos do not support a possible countermeasure from tounge bitting/pain.

posted 01-09-2008 10:39 PM            

I've got my scores.

and an OSS-3 score

Here is an experimental version of OSS-3, scoring in non-paramentric mode (using points, like we examiners use) and adding, not averaging the data. Notice that +1 at R8 produces a statistically significant result.

and OSS-2 (single issue scoring, even though it is a mixed issues exam). Notice that OSS-2 replaces the missing/zero value or Cardio at R8 on Chart 3. OSS-3 does not replace missing /zero values by default, while OSS-2 was trained with replacement data. The cummulative OSS-2 model is non-robust against missing data, while OSS-3 is an averaging model that can tollerate some missing data. It seems to me to be theoretically incorrect to replace data in mixed issues exams.

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-09-2008 10:59 PM            

------------------
"This is our hill and these are our beans."----
Leslie Nielsen as Lt. Frank Drebin, Naked Gun 1988

posted 01-10-2008 08:54 AM            

Stat and I had an interesting converation about this, and found that he got some -1 scores out of some of the pneumos, based on changes in EI ratios, whereas I found the pneumos mostly uninformative.

A quick look through the references reveals that changes in IE ratio are retained in the present DACA scoring manual (8/06 - as provided at anti), while IE ratio is not part of the pneumo scoring criteria in the ASTM standard or the Utah Guideline.

Here is Barry's Utah cheat sheet.
http://www.raymondnelson.us/qc/Utah_ZCT_Scoring_System.pdf

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-10-2008 01:27 PM         

Thanks for posting the charts. It’s nice to know how one examiner (meaning me) scores in comparison to others from time to time. I’m - curious regarding your comment “the orange tracing is another movement channel” what body parts are you recording with this channel, and is there a particular reason you have it turned down?

My scoring (with a 3 pt scale)
RQ 4: +4 ;
RQ 6: +6;
RQ 8: +5

Unfortunately Lafayette doe not support the current version of OSS-3 yet, apparently though it will be incorporated when LX-5000 debuts sometime this year. I like the idea regarding the experimental version of OSS-3, scoring in non-paramentric mode (using points, like we examiners use).

IF I only had a faster computer and high speed internet to download graphics, life would be s-o much easier- (my antiquated. computer equipment is downside).

posted 01-10-2008 02:13 PM            

We can sometimes disagree a bit.

Everytime I hear +20something I cringe. I've never seen numbers like that.

OSS-2 will produce high point totals. I think Nate Gordon's system will also produce greater numbers.

Other sources of difference involve the feature set we use.

I was incorrect earlier when I stated that changes in IE ratio are not part of ASTM (2002). It is.

Here is a cheat sheet
http://www.raymondnelson.us.qc/ASTM_Cheat_Sheet.pdf

We have to keep in mind that greater numbers do not mean greater accuracy.

Scoring/adding both pneumos will increase the numbers.

OSS-1 and OSS-2 produce higher point totals than other handscoring methods, because those methods use a uniform nonparametric model for assigning 7-position scores - meaning we expect to see as may +/3s as +/-2s and +/-1s in those models (including pneumos), while that is not the case with other handscoring methods.

An expected part of the evolutionary pareing of a maturing science is the to reduce the features of interest, as we emphasize more productive and reliable features and depricate less productive and less reliable features. You can see this in the current DACA scoring features (maybe someone could make a cheat sheet), for which the number of scorable criteria is reduced from in the past.

One thing we don't know yet is whether the reduction of number of scoreable criteria will result in a reduction of the observed variance (range or totals). There is some reason to expect a change, considering that if we have a set of 35 or 45 different criteria to score that is 35 or 45 different opportunities to assign points. Reduction to 10 or 12 features for the same data, produces a corresponding reduction in point assignment opportunities. As far as I know this has not been studied. To be sure, we could never calculate a p-value or level of significance for a spot or total score anyway - which we should do, in order to better answer our detractors. With the exception of OSS-1 and OSS-2 none of our existing systems, to my knowledge, have published descriptions of distributions of truthful and deceptive scores. Matte did attempt to provide some information in his book.

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-10-2008 02:51 PM            

Lafayette is working on OSS-3. I spoke with them about some features a couple week ago.

They are nearing completion, and might be motivated to increase the priority of the project if they heard from you.

Be sure and thank them, because they, like Limestone, have invested themselves in the project. While we have made it available to them without cost, programming and development is costly to the manufacturers.

Lafayette used this guy as a consultant.
http://www.amazon.com/Probability-Statistics-Integrated-Software-Routines/dp/0123694639/ref=sr_1_1?ie=UTF8&s=books&qid=1199990963&sr=1-1

Ronald Deep is a statistics professor from Ohio, who seems to be known about the country for his well regarded (and expensive) textbook. Just read the Key Phrases at Amazon, and you'll notice the content sounds a lot like the mathematical babble we use to describe OSS-3.

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-10-2008 04:17 PM            

Though they could just as easily be displayed in inches.

Scoring windows here are.

Pneumo = 10 seconds
EDA = 12.9 seconds
Cardio = 15 seconds

From Kircher's recomendations. I can recall the cite right now, but I bet that Barry can (tag).

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-10-2008 04:35 PM         

Yeah, I'm aware Lafayette is working on OSS-3; they were going to make an update to their software with OSS-3 and make a release a while back, but it was delayed 'for some unspecified reason.' I spoke with Chris yesterday to check the on the status, and it looks like they will release software update at the same time as they begin to offer LX-5000.

Question is- Will the LX-5000 be released before the APA National Conference or presented at the conference ? It is a competative business out there for these manufacturers. Personally, I can't wait to see how LX-5000 compares to Limestone.

As I mentioned, your experimental version OSS-3, giving the examiner the abilty to choose between p-values or generate a computerized'hand score' allowing a more direct comparison is of interest to me.

posted 01-10-2008 06:00 PM            

I wouldn't get your hopes up too high about the nonparametric point mode. Nonparametrics are really cool, and its what we've done all along with integer point assignments and arbitrary ratios. But nonparametric algorithms generally have less statistical power than their parametric alternatives. A bit of step-wise bluntness is useful when dealing with messy or irregular data that will not meet parametric requirements. The graphic above is still scored using the mean-CQ value, and three position only, using bigger-is-better and doubling the EDA values, so it can still be expected to differ from what you get by hand. Though integer-based, it is still intended to provide a p-value. At present, it works fine for single issue exams. But training it for spot scoring and mixed issues is monstrously complicated, because it requires that we know the distribution (mean and variance) of truthful and deceptive spots. That's why we need a monte carlo simulation in the absence of enough known data. It'll probably never work, but its fun to play with. The parametric OSS-3 model solved the spot distribution problem by standardizing the variance of all components and all spots, and by aggregating the data through averaging and weighted averaging, instead of simple addition. That way, the variance of the spots is effectively approximated by the variance of the total, which is not changed when we add more questions or charts, as it would be if we were to aggregate the data through addition (the way we do by hand). ARgh!

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)